{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 作业题目\n",
    "详见《readme.md》"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 导入工具和数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\ProgramData\\Anaconda3\\lib\\site-packages\\h5py\\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'channels_last'"
      ]
     },
     "execution_count": 1,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\"\"\"A very simple MNIST classifier.\n",
    "See extensive documentation at\n",
    "https://www.tensorflow.org/get_started/mnist/beginners\n",
    "\"\"\"\n",
    "from __future__ import absolute_import\n",
    "from __future__ import division\n",
    "from __future__ import print_function\n",
    "\n",
    "import argparse\n",
    "import sys\n",
    "import tensorflow as tf\n",
    "\n",
    "from tensorflow.examples.tutorials.mnist import input_data\n",
    "\n",
    "\n",
    "FLAGS = None\n",
    "#\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "#\n",
    "\n",
    "from keras.layers.core import Dense, Flatten\n",
    "from keras.layers.convolutional import Conv2D\n",
    "from keras.layers.pooling import MaxPooling2D\n",
    "\n",
    "from keras import backend as K\n",
    "from keras.objectives import categorical_crossentropy\n",
    "\n",
    "K.image_data_format() \n",
    "# C:\\ProgramData\\Anaconda3\\lib\\site-packages\\h5py\\__init__.py:36: \n",
    "# FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. \n",
    "# In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
    "# from ._conv import register_converters as _register_converters\n",
    "# Using TensorFlow backend."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们在这里调用系统提供的Mnist数据函数为我们读入数据，如果没有下载的话则进行下载。\n",
    "\n",
    "这里将data_dir改为适合你的运行环境的目录 <font color=#ff0000>** **</font>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting input_data\\train-images-idx3-ubyte.gz\n",
      "Extracting input_data\\train-labels-idx1-ubyte.gz\n",
      "Extracting input_data\\t10k-images-idx3-ubyte.gz\n",
      "Extracting input_data\\t10k-labels-idx1-ubyte.gz\n"
     ]
    }
   ],
   "source": [
    "# Import data\n",
    "data_dir = 'input_data'  #E:\\lzp\\csdn\\prj\\week6\\week6_mnist\\\n",
    "mnist = input_data.read_data_sets(data_dir, one_hot=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 熟悉样本"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " 类型是 <class 'tensorflow.contrib.learn.python.learn.datasets.base.Datasets'>\n",
      " 训练数据有 55000\n",
      " 测试数据有 10000\n",
      " 数据类型 is <class 'numpy.ndarray'>\n",
      " 标签类型 <class 'numpy.ndarray'>\n",
      " 训练集的shape (55000, 784)\n",
      " 训练集的标签的shape (55000, 10)\n",
      " 测试集的shape' is (10000, 784)\n",
      " 测试集的标签的shape (10000, 10)\n"
     ]
    }
   ],
   "source": [
    "#观察\n",
    "print (\" 类型是 %s\" % (type(mnist)))\n",
    "print (\" 训练数据有 %d\" % (mnist.train.num_examples))\n",
    "print (\" 测试数据有 %d\" % (mnist.test.num_examples))\n",
    "trainimg   = mnist.train.images\n",
    "trainlabel = mnist.train.labels\n",
    "testimg    = mnist.test.images\n",
    "testlabel  = mnist.test.labels\n",
    "# 28 * 28 * 1\n",
    "print (\" 数据类型 is %s\"    % (type(trainimg)))\n",
    "print (\" 标签类型 %s\"  % (type(trainlabel)))\n",
    "print (\" 训练集的shape %s\"   % (trainimg.shape,))\n",
    "print (\" 训练集的标签的shape %s\" % (trainlabel.shape,))\n",
    "print (\" 测试集的shape' is %s\"    % (testimg.shape,))\n",
    "print (\" 测试集的标签的shape %s\"  % (testlabel.shape,))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#样本显示\n",
    "img_show = False\n",
    "if img_show == True :\n",
    "    nsample = 1\n",
    "    # randidx = np.random.randint(trainimg.shape[0], size=nsample) \n",
    "    randidx = [1] #第n个图\n",
    "    for i in randidx:\n",
    "        curr_img   = np.reshape(trainimg[i, :], (28, 28)) # 28 by 28 matrix \n",
    "        curr_label = np.argmax(trainlabel[i, :] ) # Label\n",
    "        plt.matshow(curr_img, cmap=plt.get_cmap('gray'))\n",
    "        print (\"\" + str(i) + \"th 训练数据 \" + \"标签是 \" + str(curr_label))\n",
    "        plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 熟悉激活函数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "激活函数(Activation Functions)： "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "sigmoid tanh relu softplus RReLU  leakRelu PReLU MaxOut ELU SELU Swish cRelu MPELU"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "官网地址：http://www.tensorfly.cn/tfdoc/api_docs/python/nn.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "优缺点分析：\n",
    "1. https://zhuanlan.zhihu.com/p/22142013   \n",
    "2. https://blog.csdn.net/weixin_39881922/article/details/79045687"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "激活函数有如下分类，将在训练中起到不同作用：\n",
    "1. 可导：sigmoid、selu，不可导：relu\n",
    "2. 线性： relu、prelu,非线性:selu、elu、sigmoid、tanh\n",
    "3. 变化率：例如selu大于elu"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "#导入自定义激活函数类\n",
    "#经测试，kernel一次执行，同一类名只导入1次，如果类内容修改，需要kernel restart\n",
    "from ActivationFunction import ActivationFunction\n",
    "af = ActivationFunction()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 获取输入和标签："
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "placeholder,数值不可改变"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the model\n",
    "x = tf.placeholder(tf.float32, [None, 784])\n",
    "# 定义我们的ground truth 占位符 Define loss and optimizer\n",
    "y = tf.placeholder(tf.float32, [None, 10])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#reshape,适合图像处理\n",
    "with tf.name_scope('reshape'):\n",
    "    x_image = tf.reshape(x, [-1, 28, 28, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 熟悉keras"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 中文综述：https://blog.csdn.net/gjq246/article/details/72638343\n",
    "2. 初始化：http://keras-cn.readthedocs.io/en/latest/other/initializations/\n",
    "3. 正则化：http://keras-cn.readthedocs.io/en/latest/other/regularizers/\n",
    "4. 激活 ：http://keras-cn.readthedocs.io/en/latest/other/activations/\n",
    "5. ……"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 定义函数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 网络层"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "卷积，池化，层连接，激活，权重正则,激活函数，……"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def kerasNetNN(x,\n",
    "               filters=np.array([32,64]),\n",
    "               kernel_size=np.array([[5,5],[5,5]]),\n",
    "               strides=np.array([[5,5],[5,5]]),\n",
    "               padding='same',\n",
    "               kernel_initializer = 'glorot_uniform',  #\n",
    "               bias_initializer='zeros',\n",
    "               activation = 'relu',\n",
    "               regular = 'l2',regular_lambda = 0.0001,\n",
    "               input_shape = np.array([28,28,1]),\n",
    "               units = 1000,\n",
    "               numClasses = 10\n",
    "               ):\n",
    "#----------------------------------------------------\n",
    "# 1 功能说明：\n",
    "#   层连接，激活函数等，支持多隐层\n",
    "# 2 参数说明：\n",
    "#   filters：格式array,卷积核的数目（即输出的维度）,\n",
    "#   kernel_initializer = ['RandomNormal','RandomUniform','TruncatedNormal','VarianceScaling','Orthogonal','lecun_uniform']\n",
    "#   activation：激活函数，支持3种，'relu','selu','elu'\n",
    "#   yout:输出\n",
    "#----------------------------------------------------\n",
    "# validation of input values \n",
    "    assert len(filters) >= 1, 'filters len not less than 1 required!' \n",
    "    assert len(filters) == len(kernel_size.shape), 'kernel_size and filters size match required!'\n",
    "#---------------------------------\n",
    "    hidden_layer_num = len(filters)\n",
    "    \n",
    "    if(regular == 'l1'):\n",
    "        regularizer = keras.regularizers.l1(regular_lambda)\n",
    "    elif(regular == 'l2'):\n",
    "        regularizer = keras.regularizers.l2(regular_lambda)\n",
    "    elif(regular == 'l1_l2'):\n",
    "        regularizer = keras.regularizers.l1_l2(regular_lambda)        \n",
    "    else:\n",
    "        regularizer = None\n",
    "    #--------------------\n",
    "    # for start:\n",
    "    for i in range(hidden_layer_num):\n",
    "        #conv\n",
    "        if i == 0 :\n",
    "            net = Conv2D(filters = filters[i], kernel_size=kernel_size[i], strides=strides[i],padding=padding,\n",
    "                         kernel_initializer = kernel_initializer,bias_initializer=bias_initializer,\n",
    "                         kernel_regularizer=regularizer,bias_regularizer=regularizer,activity_regularizer=regularizer,\n",
    "                         activation=activation,\n",
    "                         input_shape=input_shape)(x)\n",
    "        else:\n",
    "            net = Conv2D(filters = filters[i], kernel_size=kernel_size[i], strides=strides[i],padding=padding,\n",
    "                         kernel_initializer = kernel_initializer,bias_initializer=bias_initializer,\n",
    "                         kernel_regularizer=regularizer,bias_regularizer=regularizer,activity_regularizer=regularizer,\n",
    "                         activation=activation)(net)\n",
    "        #pooling\n",
    "        net = MaxPooling2D(pool_size=[2,2])(net)\n",
    "    # for end\n",
    "    #--------------------    \n",
    "    net = Flatten()(net)\n",
    "    net = Dense(units=units, activation=activation)(net)\n",
    "    net = Dense(units=numClasses,activation='softmax')(net)\n",
    "    return net\n",
    "#----------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 损失计算函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def lossNN(y_pred):\n",
    "#----------------------------------------------------\n",
    "# 1 功能说明：\n",
    "#   定义损失函数，及其计算方法\n",
    "# 2 参数说明：\n",
    "#   y_pred：预测值\n",
    "#----------------------------------------------------\n",
    "    cross_entropy = tf.reduce_mean(categorical_crossentropy(y, y_pred))\n",
    "    #-----------------------\n",
    "    return cross_entropy\n",
    "#----------------------------------------------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 正则函数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "参考例程：\n",
    "1. 官网正则函数：\n",
    "  * https://www.tensorflow.org/api_guides/python/nn#Losses\n",
    "  * https://blog.csdn.net/JNingWei/article/details/77839385\n",
    "2. 可训练变量库：\n",
    "  * https://blog.csdn.net/chaowang1994/article/details/80388990"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "def regularNN(regular = 'l2',regular_lambda = 0.0001):\n",
    "#----------------------------------------------------\n",
    "# 1 功能说明：\n",
    "#   添加全局正则项\n",
    "# 2 参数说明：\n",
    "#   regular：正则函数，支持，'l2' , 'log_poisson_loss',None，没有不支持l1\n",
    "#----------------------------------------------------\n",
    "# loss_regular = tf.add_n( [tf.nn.l2_loss(w) for w in tf.get_collection(tf.GraphKeys.TRAINABLE_VARIABLES)] )\n",
    "    vars   = tf.trainable_variables() \n",
    "    if(regular == 'l2'):\n",
    "        loss_regular = tf.add_n([ tf.nn.l2_loss(v) for v in vars ]) * regular_lambda\n",
    "    elif(regular == 'log_poisson_loss'):\n",
    "        loss_regular = tf.add_n([ tf.nn.log_poisson_loss(v) for v in vars ]) * regular_lambda\n",
    "    else:\n",
    "        loss_regular = 0\n",
    "    return loss_regular\n",
    "#----------------------------------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# # 正则测试\n",
    "# RegularExpression = ['l1','l2']\n",
    "# regular_lambda = 0.0001\n",
    "# cost_regular = regularNN(W1,W2,W3,RegularExpression[1],regular_lambda)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 学习率函数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "参考例程：\n",
    "1. https://blog.csdn.net/zSean/article/details/75196092  讲理论\n",
    "2. https://blog.csdn.net/uestc_c2_403/article/details/72213286  有图示"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "def learningRateNN(learning_rate = 0.3,global_step=tf.Variable(0),decay_steps = 150,decay_rate = 0.98,staircase=True):\n",
    "#----------------------------------------------------\n",
    "# 1 功能说明：\n",
    "#   定义学习率，梯度下降\n",
    "# 2 参数说明：如下\n",
    "#----------------------------------------------------\n",
    "    #--------------学习速率的设置（学习速率呈指数下降）---------------------  \n",
    "    # tf.train.exponential_decay(learning_rate, global_, decay_steps, decay_rate, staircase=False)\n",
    "    # 公式：decayed_learning_rate=learining_rate*decay_rate^(global_step/decay_steps) \n",
    "    # learning_rate为事先设定的初始学习率；\n",
    "    # global_ = tf.Variable(0)  #必须是tf变量，且最好是0\n",
    "    # decay_steps为衰减速度。\n",
    "    # decay_rate为衰减系数；\n",
    "    # 默认值为False,当为True时，（global_step/decay_steps）则被转化为整数) ,选择不同的衰减方式，因此一般设置为Ture更好\n",
    "    eita = tf.train.exponential_decay(learning_rate,global_step,decay_steps,decay_rate,staircase) \n",
    "    return eita"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 学习率测试\n",
    "# eita =  learningRateNN(learning_rate = 0.3,global_step=tf.Variable(0),decay_steps = 150,decay_rate = 0.98,staircase=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 优化函数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. 关于global_step这个奇怪的变量：\n",
    "  * https://blog.csdn.net/leviopku/article/details/78508951  \n",
    "2. 官网优化函数： \n",
    "  * training库：https://www.tensorflow.org/api_guides/python/train  \n",
    "  * contrib库：https://www.tensorflow.org/api_docs/python/tf/contrib/opt\n",
    "  * https://www.cnblogs.com/wuzhitj/p/6648641.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def optimizerNN(eita,cost,optimizer = 'GD',momentum = 0.9, global_step=None):\n",
    "#----------------------------------------------------\n",
    "# 1 功能说明：\n",
    "#   定义优化方法，此处只使用training库，函数太多了\n",
    "# 2 参数说明：\n",
    "#----------------------------------------------------\n",
    "    if optimizer == 'GD' :\n",
    "        opti = tf.train.GradientDescentOptimizer(learning_rate = eita)\n",
    "    else optimizer == 'Adam' :\n",
    "        opti = tf.train.AdamOptimizer(learning_rate = eita,epsilon=1e-08)\n",
    "    else optimizer == 'Adadelta' :\n",
    "        opti = tf.train.AdadeltaOptimizer(learning_rate = eita,rho=0.95, epsilon=1e-08)\n",
    "    else optimizer == 'AdagradDA' :\n",
    "        opti = tf.train.AdagradDAOptimizer(learning_rate = eita,initial_accumulator_value=0.1)\n",
    "    else optimizer == 'Momentum' :\n",
    "        opti = tf.train.MomentumOptimizer(learning_rate = eita,momentum=momentum)\n",
    "    else optimizer == 'NesterovMomentum' :\n",
    "        opti = tf.train.MomentumOptimizer(learning_rate = eita,momentum=momentum, use_nesterov=True)        \n",
    "    else optimizer == 'Ftrl' :\n",
    "        opti = tf.train.FtrlOptimizer(learning_rate = eita)\n",
    "    else optimizer == 'ProximalGradientDescent' :\n",
    "        opti = tf.train.ProximalGradientDescentOptimizer(learning_rate = eita)\n",
    "    else optimizer == 'ProximalAdagrad' :\n",
    "        opti = tf.train.ProximalAdagradOptimizer(learning_rate = eita)\n",
    "    else optimizer == 'RMSProp' :\n",
    "        opti = tf.train.RMSPropOptimizer(learning_rate = eita) \n",
    "    else :# None\n",
    "        opti = tf.train.GradientDescentOptimizer(learning_rate = eita)   \n",
    "    #-----------------------\n",
    "    # 此处eita已经按照指数衰减了，因此global_step=None即可，(global_step=0时，每次调用Optimizer，global_step会自动+1)\n",
    "    # global_step = tf.train.get_or_create_global_step()\n",
    "    train_step = opti.minimize(cost,global_step=global_step) \n",
    "    return train_step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 训练函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def trainingNN(trainingIterations = 13000,batchSize = 100,\n",
    "#                #全连接网络结构参数\n",
    "#                numHiddenUnits1 = 200,numHiddenUnits2 = 80,inputSize = 784,numClasses = 10, \n",
    "               #keras网络层参数\n",
    "               filters=np.array([32,64]),\n",
    "               kernel_size=np.array([[5,5],[5,5]]),\n",
    "               strides=np.array([[5,5],[5,5]]),\n",
    "               padding='same',\n",
    "               kernel_initializer = 'glorot_uniform',\n",
    "               bias_initializer='zeros',\n",
    "               activation = 'relu',\n",
    "               regular = 'l2',regular_lambda = 0.0001,\n",
    "               input_shape = np.array([28,28,1]),\n",
    "               units = 1000,\n",
    "               numClasses = 10,\n",
    "               #初始化参数\n",
    "               initConstant = 0.1, stddevConstant = 0.1, InitializationMethod = 'gauss',               \n",
    "               #激活函数参数\n",
    "               activationFunction = 'relu',\n",
    "               #正则参数\n",
    "               RegularExpression = 'l2',  regular_lambda = 0.0001,\n",
    "               #学习率参数\n",
    "               learning_rate = 0.4,global_step=tf.Variable(0),decay_steps = 150,decay_rate = 0.98,staircase=True,\n",
    "               #损失函数参数\n",
    "               loss_calculation = 'softmax'\n",
    "               ):\n",
    "#----------------------------------------------------\n",
    "# 1 功能说明：\n",
    "#   调用初始化、激活、正则、学习率、损失计算等各函数，完成训练、精度计算、预测\n",
    "# 2 参数说明：\n",
    "#   见具体函数。\n",
    "#----------------------------------------------------\n",
    "    #网络层定义\n",
    "#     initConstant = 0.1\n",
    "#     stddevConstant = 0.1\n",
    "#     InitializationMethod = ['constant','gauss','Xavier']\n",
    "    W1,W2,W3,B1,B2,B3 = kerasNetNN(InitializationMethod,initConstant,stddevConstant,numHiddenUnits1,numHiddenUnits2,inputSize,numClasses)\n",
    "    #-----------------------\n",
    "    #激活函数\n",
    "#     activationFunction = ['relu','selu','elu']\n",
    "    #y_logits是没有激活的logits值\n",
    "    y_logits = connectNN(x,W1,W2,W3,B1,B2,B3,activationFunction = activationFunction)\n",
    "    #-----------------------\n",
    "    #正则\n",
    "#     RegularExpression = ['l1','l2']\n",
    "#     regular_lambda = 0.0001\n",
    "    cost_regular = regularNN(W1,W2,W3,RegularExpression,regular_lambda)\n",
    "    #-----------------------\n",
    "    #损失:交叉熵或mse\n",
    "    cross_entropy = lossNN(y_logits = y_logits,loss_calculation = loss_calculation)\n",
    "    #-----------------------\n",
    "    #代价\n",
    "    cost = tf.reduce_mean(cross_entropy + cost_regular)\n",
    "    # 学习率：\n",
    "#     global_step = tf.Variable(0)\n",
    "    eita =  learningRateNN(learning_rate,global_step,decay_steps,decay_rate,staircase)\n",
    "    #-----------------------\n",
    "    # 优化方法：梯度下降\n",
    "    # train_step = tf.train.GradientDescentOptimizer(learning_rate = learning_rate).minimize(cost,global_step=global_step)\n",
    "    opti = tf.train.GradientDescentOptimizer(learning_rate = eita).minimize(cost)\n",
    "    #-----------------------  \n",
    "    # 精度    \n",
    "    y_pred = tf.nn.softmax(y_logits)    \n",
    "    correct_prediction = tf.equal(tf.argmax(y_pred, 1), tf.argmax(y, 1))\n",
    "    accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))\n",
    "    #-----------------------\n",
    "    # 声明session\n",
    "    sess = tf.Session()\n",
    "    #所有变量初始化\n",
    "    init_op = tf.global_variables_initializer()\n",
    "    sess.run(init_op)\n",
    "    #-----------------------\n",
    "    for i in range(trainingIterations): #trainingIterations\n",
    "        batch_xs, batch_ys = mnist.train.next_batch(batchSize)\n",
    "        feed_dict={x: batch_xs, y: batch_ys}\n",
    "        batch_eita = sess.run(eita,feed_dict={global_step: i}) \n",
    "        batch_opti,batch_loss = sess.run([opti,cost], feed_dict=feed_dict)\n",
    "        if (i%1000 == 0) or (i == (trainingIterations-1)):\n",
    "            trainAccuracy = accuracy.eval(session=sess, feed_dict=feed_dict)\n",
    "            testAccuracy  = accuracy.eval(session=sess, feed_dict = {x: mnist.test.images,y: mnist.test.labels})\n",
    "            print (\"step=%5d, trainAccuracy=%1.2f, loss=%1.6f, eita=%1.6f,testAccuracy=%1.4f\"\n",
    "                   %(i,       trainAccuracy,     batch_loss,   batch_eita,testAccuracy      ))\n",
    "    #-------------\n",
    "#     acc = accuracy.eval(session=sess, feed_dict = {x: mnist.test.images,y: mnist.test.labels})\n",
    "#     print(\"testing accuracy=%1.4f\"%(acc))\n",
    "#----------------------------------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 训练测试\n",
    "# trainingNN()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 开始训练"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 训练参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "trainingIterations = 11000\n"
     ]
    }
   ],
   "source": [
    "#类别数量\n",
    "numClasses = 10 \n",
    "#特征维度\n",
    "inputSize = 784 \n",
    "#样本遍历次数\n",
    "epochs = 20\n",
    "#训练迭代次数\n",
    "trainingIterations = 11000 #int(trainimg.shape[0]*epochs/batchSize)\n",
    "print(\"trainingIterations = %d\"% trainingIterations)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 监测量选择说明"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "选取了随迭代变化、对性能影响大的量，打印输出包括：       \n",
    "1. step      ： 迭代序号，标记训练过程\n",
    "2. trainAccuracy：训练精度，观察变化 \n",
    "3. loss      ： 训练损失，观察是否训练不足或过拟合\n",
    "4. eita      ： 学习率，观察步进情况，了解参数的量级\n",
    "5. testAccuracy ：预测精度，观察拟合情况和预测效果"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 训练参数的影响"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "以batchSize为例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "#定义一个批次的样本数量，并训练\n",
    "# batchSize   = [10,100]\n",
    "# for i in range(len(batchSize)):\n",
    "#     print('batchSize = ' ,batchSize[i])\n",
    "#     trainingNN(batchSize = batchSize[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "小结：\n",
    "* 整体上，训练中对样本使用重复覆盖多，有助于提高精度\n",
    "* 不一定训练次数越多越好，训练少会欠拟合，训练多容易过拟合，需要根据参数和表现调试\n",
    "* 模型越复杂，相对要求的训练时间越长才能出好效果，需要在效果和时间上找到平衡"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 初始化的影响"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "InitializationMethod =  Xavier\n",
      "step=    0, trainAccuracy=0.17, loss=2.429571, eita=0.400000,testAccuracy=0.1501\n",
      "step= 1000, trainAccuracy=1.00, loss=0.146191, eita=0.354337,testAccuracy=0.9592\n",
      "step= 2000, trainAccuracy=1.00, loss=0.097048, eita=0.307609,testAccuracy=0.9753\n",
      "step= 3000, trainAccuracy=1.00, loss=0.058004, eita=0.267043,testAccuracy=0.9779\n",
      "step= 4000, trainAccuracy=1.00, loss=0.069011, eita=0.236558,testAccuracy=0.9788\n",
      "step= 5000, trainAccuracy=1.00, loss=0.047101, eita=0.205362,testAccuracy=0.9792\n",
      "step= 6000, trainAccuracy=1.00, loss=0.035372, eita=0.178280,testAccuracy=0.9800\n",
      "step= 7000, trainAccuracy=1.00, loss=0.033875, eita=0.157928,testAccuracy=0.9814\n",
      "step= 8000, trainAccuracy=1.00, loss=0.047923, eita=0.137102,testAccuracy=0.9815\n",
      "step= 9000, trainAccuracy=1.00, loss=0.032775, eita=0.119021,testAccuracy=0.9815\n",
      "step=10000, trainAccuracy=1.00, loss=0.030523, eita=0.105434,testAccuracy=0.9819\n",
      "step=11000, trainAccuracy=1.00, loss=0.028906, eita=0.091530,testAccuracy=0.9819\n",
      "step=12000, trainAccuracy=1.00, loss=0.028827, eita=0.079460,testAccuracy=0.9823\n",
      "step=12999, trainAccuracy=1.00, loss=0.027832, eita=0.070389,testAccuracy=0.9825\n",
      "InitializationMethod =  gauss , stddevConstant =  0.1\n",
      "step=    0, trainAccuracy=0.27, loss=3.141652, eita=0.400000,testAccuracy=0.2053\n",
      "step= 1000, trainAccuracy=0.99, loss=0.216906, eita=0.354337,testAccuracy=0.9609\n",
      "step= 2000, trainAccuracy=1.00, loss=0.103823, eita=0.307609,testAccuracy=0.9741\n",
      "step= 3000, trainAccuracy=1.00, loss=0.101925, eita=0.267043,testAccuracy=0.9747\n",
      "step= 4000, trainAccuracy=1.00, loss=0.097328, eita=0.236558,testAccuracy=0.9762\n",
      "step= 5000, trainAccuracy=1.00, loss=0.108738, eita=0.205362,testAccuracy=0.9798\n",
      "step= 6000, trainAccuracy=1.00, loss=0.079559, eita=0.178280,testAccuracy=0.9780\n",
      "step= 7000, trainAccuracy=1.00, loss=0.059410, eita=0.157928,testAccuracy=0.9806\n",
      "step= 8000, trainAccuracy=1.00, loss=0.058458, eita=0.137102,testAccuracy=0.9807\n",
      "step= 9000, trainAccuracy=1.00, loss=0.053001, eita=0.119021,testAccuracy=0.9815\n",
      "step=10000, trainAccuracy=1.00, loss=0.051657, eita=0.105434,testAccuracy=0.9821\n",
      "step=11000, trainAccuracy=1.00, loss=0.056265, eita=0.091530,testAccuracy=0.9804\n",
      "step=12000, trainAccuracy=1.00, loss=0.078762, eita=0.079460,testAccuracy=0.9740\n",
      "step=12999, trainAccuracy=1.00, loss=0.069644, eita=0.070389,testAccuracy=0.9759\n",
      "InitializationMethod =  constant , initConstant  =  0.1\n",
      "step=    0, trainAccuracy=0.14, loss=2.389304, eita=0.400000,testAccuracy=0.0980\n",
      "step= 1000, trainAccuracy=0.15, loss=715.209839, eita=0.354337,testAccuracy=0.0980\n",
      "step= 2000, trainAccuracy=0.06, loss=660.409241, eita=0.307609,testAccuracy=0.0980\n",
      "step= 3000, trainAccuracy=0.10, loss=609.808594, eita=0.267043,testAccuracy=0.0980\n",
      "step= 4000, trainAccuracy=0.12, loss=563.085999, eita=0.236558,testAccuracy=0.0980\n",
      "step= 5000, trainAccuracy=0.14, loss=520.001221, eita=0.205362,testAccuracy=0.0980\n",
      "step= 6000, trainAccuracy=0.03, loss=480.199890, eita=0.178280,testAccuracy=0.0980\n",
      "step= 7000, trainAccuracy=0.10, loss=443.440338, eita=0.157928,testAccuracy=0.0980\n",
      "step= 8000, trainAccuracy=0.10, loss=409.539032, eita=0.137102,testAccuracy=0.0980\n",
      "step= 9000, trainAccuracy=0.09, loss=378.218781, eita=0.119021,testAccuracy=0.0980\n",
      "step=10000, trainAccuracy=0.14, loss=349.317505, eita=0.105434,testAccuracy=0.0980\n",
      "step=11000, trainAccuracy=0.12, loss=322.635620, eita=0.091530,testAccuracy=0.0980\n",
      "step=12000, trainAccuracy=0.13, loss=298.004395, eita=0.079460,testAccuracy=0.0980\n",
      "step=12999, trainAccuracy=0.06, loss=275.294800, eita=0.070389,testAccuracy=0.0980\n"
     ]
    }
   ],
   "source": [
    "#定义初始化方法，并训练\n",
    "#   kernel_initializer = ['RandomNormal','RandomUniform','TruncatedNormal','VarianceScaling','Orthogonal','lecun_uniform']\n",
    "InitializationMethod = ['random_normal','random_uniform','TruncatedNormal','VarianceScaling','Orthogonal','lecun_uniform'] \n",
    "for i in range(1): #len(InitializationMethod)\n",
    "    print('InitializationMethod = ' ,InitializationMethod[i])\n",
    "    trainingNN(kernel_initializer = InitializationMethod[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "小结：\n",
    "* Xavier方法效果最好，收敛很快(见监测量loss)，而且比较稳定(见监测量testAccuracy)\n",
    "* gauss方法效果低于Xavier，但也可以，需要其他参数配合，防止过拟合\n",
    "* 固定值方法只在单层神经网络中验证可行，在有隐层的网络中试了多次效果很差，可以看到误差在逐渐减少，但是迭代很慢受不了，可能需要其他参数配合。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 网络结构的影响"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "隐层及神经元数目"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "numHiddenUnits1 =  100 , numHiddenUnits2 =  0\n",
      "step=    0, trainAccuracy=0.14, loss=3.424922, eita=0.400000,testAccuracy=0.1078\n",
      "step= 1000, trainAccuracy=0.99, loss=0.237386, eita=0.354337,testAccuracy=0.9590\n",
      "step= 2000, trainAccuracy=0.96, loss=0.258977, eita=0.307609,testAccuracy=0.9682\n",
      "step= 3000, trainAccuracy=0.98, loss=0.158048, eita=0.267043,testAccuracy=0.9718\n",
      "step= 4000, trainAccuracy=1.00, loss=0.075299, eita=0.236558,testAccuracy=0.9739\n",
      "step= 5000, trainAccuracy=1.00, loss=0.112960, eita=0.205362,testAccuracy=0.9769\n",
      "step= 6000, trainAccuracy=1.00, loss=0.054160, eita=0.178280,testAccuracy=0.9750\n",
      "step= 7000, trainAccuracy=1.00, loss=0.049235, eita=0.157928,testAccuracy=0.9783\n",
      "step= 8000, trainAccuracy=1.00, loss=0.091784, eita=0.137102,testAccuracy=0.9789\n",
      "step= 9000, trainAccuracy=1.00, loss=0.066709, eita=0.119021,testAccuracy=0.9777\n",
      "step=10000, trainAccuracy=1.00, loss=0.051409, eita=0.105434,testAccuracy=0.9781\n",
      "step=11000, trainAccuracy=1.00, loss=0.052146, eita=0.091530,testAccuracy=0.9786\n",
      "step=12000, trainAccuracy=1.00, loss=0.037634, eita=0.079460,testAccuracy=0.9784\n",
      "step=12999, trainAccuracy=1.00, loss=0.053306, eita=0.070389,testAccuracy=0.9785\n",
      "numHiddenUnits1 =  280 , numHiddenUnits2 =  0\n",
      "step=    0, trainAccuracy=0.20, loss=3.297922, eita=0.400000,testAccuracy=0.1208\n",
      "step= 1000, trainAccuracy=0.98, loss=0.279449, eita=0.354337,testAccuracy=0.9612\n",
      "step= 2000, trainAccuracy=0.98, loss=0.214556, eita=0.307609,testAccuracy=0.9739\n",
      "step= 3000, trainAccuracy=1.00, loss=0.168815, eita=0.267043,testAccuracy=0.9778\n",
      "step= 4000, trainAccuracy=1.00, loss=0.095884, eita=0.236558,testAccuracy=0.9749\n",
      "step= 5000, trainAccuracy=1.00, loss=0.115544, eita=0.205362,testAccuracy=0.9791\n",
      "step= 6000, trainAccuracy=1.00, loss=0.088438, eita=0.178280,testAccuracy=0.9798\n",
      "step= 7000, trainAccuracy=1.00, loss=0.084696, eita=0.157928,testAccuracy=0.9808\n",
      "step= 8000, trainAccuracy=1.00, loss=0.072764, eita=0.137102,testAccuracy=0.9798\n",
      "step= 9000, trainAccuracy=1.00, loss=0.074638, eita=0.119021,testAccuracy=0.9804\n",
      "step=10000, trainAccuracy=1.00, loss=0.066679, eita=0.105434,testAccuracy=0.9811\n",
      "step=11000, trainAccuracy=1.00, loss=0.073139, eita=0.091530,testAccuracy=0.9812\n",
      "step=12000, trainAccuracy=1.00, loss=0.065914, eita=0.079460,testAccuracy=0.9815\n",
      "step=12999, trainAccuracy=1.00, loss=0.059803, eita=0.070389,testAccuracy=0.9809\n",
      "numHiddenUnits1 =  200 , numHiddenUnits2 =  80\n",
      "step=    0, trainAccuracy=0.23, loss=3.512236, eita=0.400000,testAccuracy=0.1475\n",
      "step= 1000, trainAccuracy=0.99, loss=0.154811, eita=0.354337,testAccuracy=0.9624\n",
      "step= 2000, trainAccuracy=1.00, loss=0.154837, eita=0.307609,testAccuracy=0.9704\n",
      "step= 3000, trainAccuracy=1.00, loss=0.098287, eita=0.267043,testAccuracy=0.9726\n",
      "step= 4000, trainAccuracy=0.99, loss=0.098824, eita=0.236558,testAccuracy=0.9751\n",
      "step= 5000, trainAccuracy=1.00, loss=0.074335, eita=0.205362,testAccuracy=0.9753\n",
      "step= 6000, trainAccuracy=1.00, loss=0.064894, eita=0.178280,testAccuracy=0.9817\n",
      "step= 7000, trainAccuracy=1.00, loss=0.075974, eita=0.157928,testAccuracy=0.9809\n",
      "step= 8000, trainAccuracy=1.00, loss=0.057729, eita=0.137102,testAccuracy=0.9819\n",
      "step= 9000, trainAccuracy=1.00, loss=0.056534, eita=0.119021,testAccuracy=0.9815\n",
      "step=10000, trainAccuracy=1.00, loss=0.052044, eita=0.105434,testAccuracy=0.9821\n",
      "step=11000, trainAccuracy=1.00, loss=0.048793, eita=0.091530,testAccuracy=0.9823\n",
      "step=12000, trainAccuracy=1.00, loss=0.046599, eita=0.079460,testAccuracy=0.9817\n",
      "step=12999, trainAccuracy=1.00, loss=0.045531, eita=0.070389,testAccuracy=0.9827\n"
     ]
    }
   ],
   "source": [
    "#设置网络结构，并训练\n",
    "numHiddenUnits = [[100,  0], #第1种结构的第1、2隐层的神经元数目,0表示没有该隐层\n",
    "                  [280,  0], #第2种结构的第1、2隐层的神经元数目,0表示没有该隐层\n",
    "                  [200, 80]] #第3种结构的第1、2隐层的神经元数目\n",
    "for i in range(len(numHiddenUnits)):\n",
    "# for i in range(1,2):\n",
    "    numHiddenUnits1 = numHiddenUnits[i][0] \n",
    "    numHiddenUnits2 = numHiddenUnits[i][1] \n",
    "    print('numHiddenUnits1 = ',numHiddenUnits1,', numHiddenUnits2 = ',numHiddenUnits2)\n",
    "    trainingNN(numHiddenUnits1 = numHiddenUnits1, numHiddenUnits2 = numHiddenUnits2)\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "小结：\n",
    "* 隐层数目相同，神经元数目增加，有助于获得更好精度\n",
    "* 神经元数目相同，优先设置隐层数目多的结构，预测效果一般更好。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 激活函数的影响"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "activationFunction =  relu\n",
      "step=    0, trainAccuracy=0.13, loss=2.839070, eita=0.200000,testAccuracy=0.1022\n",
      "step= 1000, trainAccuracy=0.99, loss=0.181809, eita=0.106288,testAccuracy=0.9554\n",
      "step= 2000, trainAccuracy=0.99, loss=0.207884, eita=0.050837,testAccuracy=0.9689\n",
      "step= 3000, trainAccuracy=1.00, loss=0.104175, eita=0.024315,testAccuracy=0.9735\n",
      "step= 4000, trainAccuracy=1.00, loss=0.089707, eita=0.012922,testAccuracy=0.9717\n",
      "step= 5000, trainAccuracy=1.00, loss=0.100331, eita=0.006181,testAccuracy=0.9764\n",
      "step= 6000, trainAccuracy=1.00, loss=0.085677, eita=0.002956,testAccuracy=0.9755\n",
      "step= 7000, trainAccuracy=1.00, loss=0.081749, eita=0.001571,testAccuracy=0.9767\n",
      "step= 8000, trainAccuracy=1.00, loss=0.069027, eita=0.000751,testAccuracy=0.9785\n",
      "step= 9000, trainAccuracy=1.00, loss=0.067764, eita=0.000359,testAccuracy=0.9783\n",
      "step=10000, trainAccuracy=1.00, loss=0.066035, eita=0.000191,testAccuracy=0.9791\n",
      "step=11000, trainAccuracy=1.00, loss=0.064262, eita=0.000091,testAccuracy=0.9801\n",
      "step=12000, trainAccuracy=1.00, loss=0.060243, eita=0.000044,testAccuracy=0.9790\n",
      "step=12999, trainAccuracy=1.00, loss=0.058429, eita=0.000023,testAccuracy=0.9788\n",
      "activationFunction =  selu\n",
      "step=    0, trainAccuracy=0.17, loss=3.314286, eita=0.200000,testAccuracy=0.1174\n",
      "step= 1000, trainAccuracy=1.00, loss=0.136967, eita=0.106288,testAccuracy=0.9599\n",
      "step= 2000, trainAccuracy=0.99, loss=0.177520, eita=0.050837,testAccuracy=0.9691\n",
      "step= 3000, trainAccuracy=1.00, loss=0.079219, eita=0.024315,testAccuracy=0.9707\n",
      "step= 4000, trainAccuracy=1.00, loss=0.135271, eita=0.012922,testAccuracy=0.9734\n",
      "step= 5000, trainAccuracy=1.00, loss=0.088622, eita=0.006181,testAccuracy=0.9748\n",
      "step= 6000, trainAccuracy=1.00, loss=0.087459, eita=0.002956,testAccuracy=0.9753\n",
      "step= 7000, trainAccuracy=1.00, loss=0.078246, eita=0.001571,testAccuracy=0.9770\n",
      "step= 8000, trainAccuracy=1.00, loss=0.085734, eita=0.000751,testAccuracy=0.9766\n",
      "step= 9000, trainAccuracy=1.00, loss=0.086188, eita=0.000359,testAccuracy=0.9787\n",
      "step=10000, trainAccuracy=1.00, loss=0.070580, eita=0.000191,testAccuracy=0.9773\n",
      "step=11000, trainAccuracy=1.00, loss=0.074412, eita=0.000091,testAccuracy=0.9790\n",
      "step=12000, trainAccuracy=1.00, loss=0.067677, eita=0.000044,testAccuracy=0.9797\n",
      "step=12999, trainAccuracy=1.00, loss=0.062745, eita=0.000023,testAccuracy=0.9785\n",
      "activationFunction =  elu\n",
      "step=    0, trainAccuracy=0.14, loss=3.223276, eita=0.200000,testAccuracy=0.0980\n",
      "step= 1000, trainAccuracy=1.00, loss=0.173497, eita=0.106288,testAccuracy=0.9575\n",
      "step= 2000, trainAccuracy=0.98, loss=0.222858, eita=0.050837,testAccuracy=0.9671\n",
      "step= 3000, trainAccuracy=0.99, loss=0.156366, eita=0.024315,testAccuracy=0.9730\n",
      "step= 4000, trainAccuracy=1.00, loss=0.113447, eita=0.012922,testAccuracy=0.9763\n",
      "step= 5000, trainAccuracy=1.00, loss=0.151421, eita=0.006181,testAccuracy=0.9753\n",
      "step= 6000, trainAccuracy=0.99, loss=0.205242, eita=0.002956,testAccuracy=0.9747\n",
      "step= 7000, trainAccuracy=1.00, loss=0.077475, eita=0.001571,testAccuracy=0.9780\n",
      "step= 8000, trainAccuracy=1.00, loss=0.100265, eita=0.000751,testAccuracy=0.9790\n",
      "step= 9000, trainAccuracy=1.00, loss=0.067679, eita=0.000359,testAccuracy=0.9801\n",
      "step=10000, trainAccuracy=1.00, loss=0.068655, eita=0.000191,testAccuracy=0.9790\n",
      "step=11000, trainAccuracy=1.00, loss=0.066749, eita=0.000091,testAccuracy=0.9760\n",
      "step=12000, trainAccuracy=1.00, loss=0.067178, eita=0.000044,testAccuracy=0.9800\n",
      "step=12999, trainAccuracy=1.00, loss=0.069885, eita=0.000023,testAccuracy=0.9800\n"
     ]
    }
   ],
   "source": [
    "#定义激活函数，并训练\n",
    "activationFunction = ['relu','selu','elu']\n",
    "for i in range(len(activationFunction)):\n",
    "    print('activationFunction = ' ,activationFunction[i])\n",
    "    trainingNN(activationFunction = activationFunction[i],learning_rate = 0.2 , decay_rate = 0.9)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "小结：\n",
    "* relu作为基本的激活函数，收敛快，表现稳定\n",
    "* selu收敛最快，但是经常会跑飞。通过查看激活函数图形，可以看到selu斜率最大，因此需要其他参数例如较小的学习率等配合，发挥selu的效果\n",
    "* elu收敛快，且收敛平稳，大多数情况下效果均衡，这可能与elu可导同时斜率适中有关"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 学习率的影响"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "初始学习率和步进下降系数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "learningRate =  0.5 decay_rate =  0.93\n",
      "step=    0, trainAccuracy=0.17, loss=3.287596, eita=0.500000,testAccuracy=0.1010\n",
      "step= 1000, trainAccuracy=1.00, loss=0.133813, eita=0.323495,testAccuracy=0.9660\n",
      "step= 2000, trainAccuracy=0.99, loss=0.218233, eita=0.194647,testAccuracy=0.9728\n",
      "step= 3000, trainAccuracy=1.00, loss=0.075372, eita=0.117119,testAccuracy=0.9783\n",
      "step= 4000, trainAccuracy=1.00, loss=0.087444, eita=0.075775,testAccuracy=0.9766\n",
      "step= 5000, trainAccuracy=1.00, loss=0.065243, eita=0.045594,testAccuracy=0.9768\n",
      "step= 6000, trainAccuracy=1.00, loss=0.064249, eita=0.027434,testAccuracy=0.9772\n",
      "step= 7000, trainAccuracy=1.00, loss=0.076648, eita=0.017749,testAccuracy=0.9799\n",
      "step= 8000, trainAccuracy=1.00, loss=0.053872, eita=0.010680,testAccuracy=0.9804\n",
      "step= 9000, trainAccuracy=1.00, loss=0.049121, eita=0.006426,testAccuracy=0.9803\n",
      "step=10000, trainAccuracy=1.00, loss=0.046307, eita=0.004158,testAccuracy=0.9814\n",
      "step=11000, trainAccuracy=1.00, loss=0.043494, eita=0.002502,testAccuracy=0.9807\n",
      "step=12000, trainAccuracy=1.00, loss=0.041224, eita=0.001505,testAccuracy=0.9818\n",
      "step=12999, trainAccuracy=1.00, loss=0.038302, eita=0.000974,testAccuracy=0.9816\n",
      "learningRate =  0.5 decay_rate =  0.98\n",
      "step=    0, trainAccuracy=0.16, loss=3.136833, eita=0.500000,testAccuracy=0.1028\n",
      "step= 1000, trainAccuracy=0.99, loss=0.211238, eita=0.442921,testAccuracy=0.9648\n",
      "step= 2000, trainAccuracy=1.00, loss=0.079709, eita=0.384511,testAccuracy=0.9724\n",
      "step= 3000, trainAccuracy=1.00, loss=0.083697, eita=0.333804,testAccuracy=0.9761\n",
      "step= 4000, trainAccuracy=1.00, loss=0.083767, eita=0.295698,testAccuracy=0.9714\n",
      "step= 5000, trainAccuracy=1.00, loss=0.069436, eita=0.256703,testAccuracy=0.9754\n",
      "step= 6000, trainAccuracy=1.00, loss=0.064194, eita=0.222850,testAccuracy=0.9755\n",
      "step= 7000, trainAccuracy=1.00, loss=0.065439, eita=0.197410,testAccuracy=0.9784\n",
      "step= 8000, trainAccuracy=1.00, loss=0.056051, eita=0.171377,testAccuracy=0.9799\n",
      "step= 9000, trainAccuracy=0.95, loss=0.457769, eita=0.148777,testAccuracy=0.9269\n",
      "step=10000, trainAccuracy=1.00, loss=0.412858, eita=0.131793,testAccuracy=0.9585\n",
      "step=11000, trainAccuracy=1.00, loss=0.199146, eita=0.114413,testAccuracy=0.9735\n",
      "step=12000, trainAccuracy=1.00, loss=0.158622, eita=0.099325,testAccuracy=0.9752\n",
      "step=12999, trainAccuracy=1.00, loss=0.147737, eita=0.087986,testAccuracy=0.9766\n",
      "learningRate =  0.3 decay_rate =  0.93\n",
      "step=    0, trainAccuracy=0.15, loss=3.073013, eita=0.300000,testAccuracy=0.1352\n",
      "step= 1000, trainAccuracy=0.99, loss=0.250618, eita=0.194097,testAccuracy=0.9579\n",
      "step= 2000, trainAccuracy=1.00, loss=0.148008, eita=0.116788,testAccuracy=0.9680\n",
      "step= 3000, trainAccuracy=1.00, loss=0.097503, eita=0.070272,testAccuracy=0.9763\n",
      "step= 4000, trainAccuracy=0.99, loss=0.160483, eita=0.045465,testAccuracy=0.9752\n",
      "step= 5000, trainAccuracy=1.00, loss=0.074934, eita=0.027356,testAccuracy=0.9798\n",
      "step= 6000, trainAccuracy=1.00, loss=0.091275, eita=0.016460,testAccuracy=0.9789\n",
      "step= 7000, trainAccuracy=1.00, loss=0.070083, eita=0.010650,testAccuracy=0.9788\n",
      "step= 8000, trainAccuracy=1.00, loss=0.062917, eita=0.006408,testAccuracy=0.9799\n",
      "step= 9000, trainAccuracy=1.00, loss=0.061041, eita=0.003856,testAccuracy=0.9802\n",
      "step=10000, trainAccuracy=1.00, loss=0.055718, eita=0.002495,testAccuracy=0.9791\n",
      "step=11000, trainAccuracy=1.00, loss=0.055897, eita=0.001501,testAccuracy=0.9802\n",
      "step=12000, trainAccuracy=1.00, loss=0.053837, eita=0.000903,testAccuracy=0.9810\n",
      "step=12999, trainAccuracy=1.00, loss=0.051260, eita=0.000584,testAccuracy=0.9804\n",
      "learningRate =  0.3 decay_rate =  0.98\n",
      "step=    0, trainAccuracy=0.18, loss=3.317314, eita=0.300000,testAccuracy=0.0958\n",
      "step= 1000, trainAccuracy=1.00, loss=0.104660, eita=0.265753,testAccuracy=0.9608\n",
      "step= 2000, trainAccuracy=1.00, loss=0.160224, eita=0.230707,testAccuracy=0.9703\n",
      "step= 3000, trainAccuracy=1.00, loss=0.097119, eita=0.200282,testAccuracy=0.9736\n",
      "step= 4000, trainAccuracy=1.00, loss=0.101852, eita=0.177419,testAccuracy=0.9777\n",
      "step= 5000, trainAccuracy=1.00, loss=0.079106, eita=0.154022,testAccuracy=0.9766\n",
      "step= 6000, trainAccuracy=1.00, loss=0.072992, eita=0.133710,testAccuracy=0.9789\n",
      "step= 7000, trainAccuracy=1.00, loss=0.076564, eita=0.118446,testAccuracy=0.9778\n",
      "step= 8000, trainAccuracy=1.00, loss=0.062947, eita=0.102826,testAccuracy=0.9794\n",
      "step= 9000, trainAccuracy=1.00, loss=0.060245, eita=0.089266,testAccuracy=0.9802\n",
      "step=10000, trainAccuracy=1.00, loss=0.057090, eita=0.079076,testAccuracy=0.9796\n",
      "step=11000, trainAccuracy=1.00, loss=0.057495, eita=0.068648,testAccuracy=0.9793\n",
      "step=12000, trainAccuracy=1.00, loss=0.053748, eita=0.059595,testAccuracy=0.9805\n",
      "step=12999, trainAccuracy=1.00, loss=0.053492, eita=0.052792,testAccuracy=0.9800\n"
     ]
    }
   ],
   "source": [
    "#定义学习率，并训练\n",
    "learningRate = [0.5,0.3]\n",
    "decay_rate   = [0.93,0.98]\n",
    "for i in range(len(learningRate)):\n",
    "    for j in range(len(decay_rate)):\n",
    "        print('learningRate = ' ,learningRate[i],'decay_rate = ' ,decay_rate[j])\n",
    "        trainingNN(learning_rate = learningRate[i],decay_rate = decay_rate[j])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "小结：\n",
    "* 学习率是本次调参消耗精力最大的一个参数，与训练迭代参数、批次参数、网络结构、激活函数、正则等关系都很密切，很难调\n",
    "* 初始值：网络结构复杂/批次大时，适于较大值，便于快速快速收敛，同时辅助较大下降系数，避免过拟合；激活函数收敛快(例如selu)时，需要配小值\n",
    "* 下降值：配合初始值使用，即保证一定收敛速度，又不过拟合\n",
    "* 步进decay_steps：调优过程中也是一个重要参数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 正则表达式的影响"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "RegularExpression =  l1 lambda =  0.001\n",
      "step=    0, trainAccuracy=0.14, loss=15.752938, eita=0.400000,testAccuracy=0.1315\n",
      "step= 1000, trainAccuracy=0.94, loss=0.900204, eita=0.354337,testAccuracy=0.9213\n",
      "step= 2000, trainAccuracy=0.97, loss=0.683569, eita=0.307609,testAccuracy=0.9323\n",
      "step= 3000, trainAccuracy=0.95, loss=0.625435, eita=0.267043,testAccuracy=0.9289\n",
      "step= 4000, trainAccuracy=0.89, loss=0.974937, eita=0.236558,testAccuracy=0.9058\n",
      "step= 5000, trainAccuracy=0.97, loss=0.670109, eita=0.205362,testAccuracy=0.9225\n",
      "step= 6000, trainAccuracy=0.96, loss=0.670483, eita=0.178280,testAccuracy=0.9271\n",
      "step= 7000, trainAccuracy=0.97, loss=0.606056, eita=0.157928,testAccuracy=0.9283\n",
      "step= 8000, trainAccuracy=0.98, loss=0.525477, eita=0.137102,testAccuracy=0.9474\n",
      "step= 9000, trainAccuracy=0.97, loss=0.603286, eita=0.119021,testAccuracy=0.9227\n",
      "step=10000, trainAccuracy=0.68, loss=1.735352, eita=0.105434,testAccuracy=0.7413\n",
      "step=11000, trainAccuracy=0.96, loss=0.549732, eita=0.091530,testAccuracy=0.9391\n",
      "step=12000, trainAccuracy=0.96, loss=0.568546, eita=0.079460,testAccuracy=0.9228\n",
      "step=12999, trainAccuracy=0.99, loss=0.532393, eita=0.070389,testAccuracy=0.9218\n",
      "RegularExpression =  l1 lambda =  0.0001\n",
      "step=    0, trainAccuracy=0.15, loss=4.422222, eita=0.400000,testAccuracy=0.1220\n",
      "step= 1000, trainAccuracy=1.00, loss=0.872496, eita=0.354337,testAccuracy=0.9559\n",
      "step= 2000, trainAccuracy=1.00, loss=0.498749, eita=0.307609,testAccuracy=0.9699\n",
      "step= 3000, trainAccuracy=0.99, loss=0.362984, eita=0.267043,testAccuracy=0.9725\n",
      "step= 4000, trainAccuracy=1.00, loss=0.257116, eita=0.236558,testAccuracy=0.9723\n",
      "step= 5000, trainAccuracy=1.00, loss=0.235602, eita=0.205362,testAccuracy=0.9750\n",
      "step= 6000, trainAccuracy=1.00, loss=0.242407, eita=0.178280,testAccuracy=0.9690\n",
      "step= 7000, trainAccuracy=0.99, loss=0.356472, eita=0.157928,testAccuracy=0.9632\n",
      "step= 8000, trainAccuracy=1.00, loss=0.259015, eita=0.137102,testAccuracy=0.9720\n",
      "step= 9000, trainAccuracy=1.00, loss=0.255485, eita=0.119021,testAccuracy=0.9726\n",
      "step=10000, trainAccuracy=0.99, loss=0.287285, eita=0.105434,testAccuracy=0.9747\n",
      "step=11000, trainAccuracy=1.00, loss=0.172007, eita=0.091530,testAccuracy=0.9741\n",
      "step=12000, trainAccuracy=1.00, loss=0.200920, eita=0.079460,testAccuracy=0.9734\n",
      "step=12999, trainAccuracy=1.00, loss=0.179376, eita=0.070389,testAccuracy=0.9726\n",
      "RegularExpression =  l2 lambda =  0.001\n",
      "step=    0, trainAccuracy=0.17, loss=4.147881, eita=0.400000,testAccuracy=0.1008\n",
      "step= 1000, trainAccuracy=0.99, loss=0.553669, eita=0.354337,testAccuracy=0.9644\n",
      "step= 2000, trainAccuracy=1.00, loss=0.276826, eita=0.307609,testAccuracy=0.9696\n",
      "step= 3000, trainAccuracy=0.99, loss=0.222228, eita=0.267043,testAccuracy=0.9747\n",
      "step= 4000, trainAccuracy=1.00, loss=0.197915, eita=0.236558,testAccuracy=0.9748\n",
      "step= 5000, trainAccuracy=1.00, loss=0.211716, eita=0.205362,testAccuracy=0.9736\n",
      "step= 6000, trainAccuracy=1.00, loss=0.197394, eita=0.178280,testAccuracy=0.9734\n",
      "step= 7000, trainAccuracy=1.00, loss=0.121743, eita=0.157928,testAccuracy=0.9769\n",
      "step= 8000, trainAccuracy=1.00, loss=0.155757, eita=0.137102,testAccuracy=0.9788\n",
      "step= 9000, trainAccuracy=0.99, loss=0.159078, eita=0.119021,testAccuracy=0.9668\n",
      "step=10000, trainAccuracy=1.00, loss=0.115324, eita=0.105434,testAccuracy=0.9802\n",
      "step=11000, trainAccuracy=0.99, loss=0.139341, eita=0.091530,testAccuracy=0.9756\n",
      "step=12000, trainAccuracy=1.00, loss=0.169381, eita=0.079460,testAccuracy=0.9765\n",
      "step=12999, trainAccuracy=1.00, loss=0.126446, eita=0.070389,testAccuracy=0.9782\n",
      "RegularExpression =  l2 lambda =  0.0001\n",
      "step=    0, trainAccuracy=0.16, loss=3.213193, eita=0.400000,testAccuracy=0.1088\n",
      "step= 1000, trainAccuracy=1.00, loss=0.172484, eita=0.354337,testAccuracy=0.9647\n",
      "step= 2000, trainAccuracy=1.00, loss=0.094474, eita=0.307609,testAccuracy=0.9743\n",
      "step= 3000, trainAccuracy=1.00, loss=0.092723, eita=0.267043,testAccuracy=0.9744\n",
      "step= 4000, trainAccuracy=1.00, loss=0.119653, eita=0.236558,testAccuracy=0.9757\n",
      "step= 5000, trainAccuracy=1.00, loss=0.078554, eita=0.205362,testAccuracy=0.9783\n",
      "step= 6000, trainAccuracy=1.00, loss=0.074039, eita=0.178280,testAccuracy=0.9801\n",
      "step= 7000, trainAccuracy=1.00, loss=0.059545, eita=0.157928,testAccuracy=0.9796\n",
      "step= 8000, trainAccuracy=1.00, loss=0.055604, eita=0.137102,testAccuracy=0.9797\n",
      "step= 9000, trainAccuracy=1.00, loss=0.054083, eita=0.119021,testAccuracy=0.9807\n",
      "step=10000, trainAccuracy=1.00, loss=0.051544, eita=0.105434,testAccuracy=0.9800\n",
      "step=11000, trainAccuracy=1.00, loss=0.050170, eita=0.091530,testAccuracy=0.9800\n",
      "step=12000, trainAccuracy=1.00, loss=0.047771, eita=0.079460,testAccuracy=0.9806\n",
      "step=12999, trainAccuracy=1.00, loss=0.042442, eita=0.070389,testAccuracy=0.9807\n"
     ]
    }
   ],
   "source": [
    "#定义正则表达式，并训练\n",
    "RegularExpression = ['l1','l2']\n",
    "regular_lambda = [0.001,0.0001]\n",
    "for i in range(len(RegularExpression)):\n",
    "    for j in range(len(regular_lambda)):\n",
    "        print('RegularExpression = ' ,RegularExpression[i],'lambda = ',regular_lambda[j])\n",
    "        trainingNN(RegularExpression = RegularExpression[i],regular_lambda = regular_lambda[j])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "小结：\n",
    "* 正则作用极其重要，尤其当网络结构复杂时，正则项是必备项，避免还未找到极值点就过拟合，保证训练稳定收敛不跑飞,而且所有W都要正则不能漏\n",
    "* l1正则：能够实现稀疏，但是有时会收敛过程会震荡，且惩罚不够有力\n",
    "* l2正则：惩罚力度大，比较平稳，本次任务表现比较好\n",
    "* 惩罚系数lambda：最重要的调参工作之一，需要配合网络复杂程度、迭代次数等调试,例如结构复杂时惩罚加大。例如l2情况下，本次任务前期lambda=1e-5表现最好，后期1e-4表现好，还有很大发挥空间"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 损失计算的影响"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loss_calculation =  manual\n",
      "step=    0, trainAccuracy=0.19, loss=2.937502, eita=0.400000,testAccuracy=0.1283\n",
      "step= 1000, trainAccuracy=0.99, loss=0.279545, eita=0.354337,testAccuracy=0.9667\n",
      "step= 2000, trainAccuracy=1.00, loss=0.086593, eita=0.307609,testAccuracy=0.9743\n",
      "step= 3000, trainAccuracy=1.00, loss=0.131396, eita=0.267043,testAccuracy=0.9729\n",
      "step= 4000, trainAccuracy=1.00, loss=0.078062, eita=0.236558,testAccuracy=0.9788\n",
      "step= 5000, trainAccuracy=1.00, loss=0.087291, eita=0.205362,testAccuracy=0.9797\n",
      "step= 6000, trainAccuracy=1.00, loss=0.063167, eita=0.178280,testAccuracy=0.9802\n",
      "step= 7000, trainAccuracy=1.00, loss=0.076729, eita=0.157928,testAccuracy=0.9803\n",
      "step= 8000, trainAccuracy=1.00, loss=0.056848, eita=0.137102,testAccuracy=0.9816\n",
      "step= 9000, trainAccuracy=1.00, loss=0.055274, eita=0.119021,testAccuracy=0.9819\n",
      "step=10000, trainAccuracy=1.00, loss=0.051767, eita=0.105434,testAccuracy=0.9818\n",
      "step=11000, trainAccuracy=1.00, loss=0.047683, eita=0.091530,testAccuracy=0.9817\n",
      "step=12000, trainAccuracy=1.00, loss=0.046114, eita=0.079460,testAccuracy=0.9822\n",
      "step=12999, trainAccuracy=1.00, loss=0.042422, eita=0.070389,testAccuracy=0.9814\n",
      "loss_calculation =  softmax\n",
      "step=    0, trainAccuracy=0.12, loss=3.245414, eita=0.400000,testAccuracy=0.1032\n",
      "step= 1000, trainAccuracy=0.99, loss=0.146358, eita=0.354337,testAccuracy=0.9650\n",
      "step= 2000, trainAccuracy=1.00, loss=0.126295, eita=0.307609,testAccuracy=0.9724\n",
      "step= 3000, trainAccuracy=1.00, loss=0.079250, eita=0.267043,testAccuracy=0.9740\n",
      "step= 4000, trainAccuracy=1.00, loss=0.092532, eita=0.236558,testAccuracy=0.9766\n",
      "step= 5000, trainAccuracy=1.00, loss=0.089355, eita=0.205362,testAccuracy=0.9784\n",
      "step= 6000, trainAccuracy=1.00, loss=0.076925, eita=0.178280,testAccuracy=0.9784\n",
      "step= 7000, trainAccuracy=1.00, loss=0.077080, eita=0.157928,testAccuracy=0.9781\n",
      "step= 8000, trainAccuracy=1.00, loss=0.062736, eita=0.137102,testAccuracy=0.9792\n",
      "step= 9000, trainAccuracy=1.00, loss=0.052994, eita=0.119021,testAccuracy=0.9800\n",
      "step=10000, trainAccuracy=1.00, loss=0.052993, eita=0.105434,testAccuracy=0.9789\n",
      "step=11000, trainAccuracy=1.00, loss=0.049847, eita=0.091530,testAccuracy=0.9802\n",
      "step=12000, trainAccuracy=1.00, loss=0.045834, eita=0.079460,testAccuracy=0.9798\n",
      "step=12999, trainAccuracy=1.00, loss=0.045520, eita=0.070389,testAccuracy=0.9802\n",
      "loss_calculation =  mse\n",
      "step=    0, trainAccuracy=0.12, loss=0.169428, eita=0.400000,testAccuracy=0.1374\n",
      "step= 1000, trainAccuracy=0.93, loss=0.075342, eita=0.354337,testAccuracy=0.8953\n",
      "step= 2000, trainAccuracy=0.96, loss=0.068067, eita=0.307609,testAccuracy=0.9191\n",
      "step= 3000, trainAccuracy=0.91, loss=0.068845, eita=0.267043,testAccuracy=0.9283\n",
      "step= 4000, trainAccuracy=0.92, loss=0.064797, eita=0.236558,testAccuracy=0.9341\n",
      "step= 5000, trainAccuracy=0.96, loss=0.054740, eita=0.205362,testAccuracy=0.9397\n",
      "step= 6000, trainAccuracy=0.96, loss=0.054885, eita=0.178280,testAccuracy=0.9416\n",
      "step= 7000, trainAccuracy=0.99, loss=0.045530, eita=0.157928,testAccuracy=0.9462\n",
      "step= 8000, trainAccuracy=0.99, loss=0.042012, eita=0.137102,testAccuracy=0.9488\n",
      "step= 9000, trainAccuracy=0.95, loss=0.044468, eita=0.119021,testAccuracy=0.9533\n",
      "step=10000, trainAccuracy=0.96, loss=0.041500, eita=0.105434,testAccuracy=0.9549\n",
      "step=11000, trainAccuracy=0.99, loss=0.036973, eita=0.091530,testAccuracy=0.9568\n",
      "step=12000, trainAccuracy=0.96, loss=0.036866, eita=0.079460,testAccuracy=0.9601\n",
      "step=12999, trainAccuracy=0.98, loss=0.032940, eita=0.070389,testAccuracy=0.9591\n"
     ]
    }
   ],
   "source": [
    "#定义损失计算方式，并训练\n",
    "loss_calculation = ['manual','softmax','mse'] \n",
    "for i in range(len(loss_calculation)):\n",
    "    print('loss_calculation = ' ,loss_calculation[i])\n",
    "    trainingNN(loss_calculation = loss_calculation[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "小结：\n",
    "* softmax几乎是输出层激活函数的唯一选择，收敛快，求导后表达式很简单,计算很快，因此目前还没有找到比softmax更好的函数\n",
    "* (1) manual部分：softmax手动计算方式，目前表现尚好，还没出现问题，课件中提到有数值问题，因此有待进一步验证，不作为默认计算方式\n",
    "* (2) softmax部分：softmax使用系统函数计算方式，表现稳定，因此作为默认的计算方式\n",
    "* (3) mse部分：使用MSE作为损失函数，有一定合理性，效果不及softmax"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 参数合影"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "参数逐个检查，记录最优"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "step=    0, trainAccuracy=0.10, loss=2.943863, eita=0.400000,testAccuracy=0.1008\n",
      "step= 1000, trainAccuracy=0.99, loss=0.148374, eita=0.354337,testAccuracy=0.9605\n",
      "step= 2000, trainAccuracy=1.00, loss=0.113335, eita=0.307609,testAccuracy=0.9754\n",
      "step= 3000, trainAccuracy=1.00, loss=0.113394, eita=0.267043,testAccuracy=0.9743\n",
      "step= 4000, trainAccuracy=1.00, loss=0.081421, eita=0.236558,testAccuracy=0.9773\n",
      "step= 5000, trainAccuracy=1.00, loss=0.072134, eita=0.205362,testAccuracy=0.9793\n",
      "step= 6000, trainAccuracy=1.00, loss=0.066714, eita=0.178280,testAccuracy=0.9795\n",
      "step= 7000, trainAccuracy=1.00, loss=0.104032, eita=0.157928,testAccuracy=0.9779\n",
      "step= 8000, trainAccuracy=1.00, loss=0.057324, eita=0.137102,testAccuracy=0.9816\n",
      "step= 9000, trainAccuracy=1.00, loss=0.056135, eita=0.119021,testAccuracy=0.9818\n",
      "step=10000, trainAccuracy=1.00, loss=0.053918, eita=0.105434,testAccuracy=0.9813\n",
      "step=11000, trainAccuracy=1.00, loss=0.047661, eita=0.091530,testAccuracy=0.9818\n",
      "step=12000, trainAccuracy=1.00, loss=0.046253, eita=0.079460,testAccuracy=0.9825\n",
      "step=12999, trainAccuracy=1.00, loss=0.044403, eita=0.070389,testAccuracy=0.9820\n"
     ]
    }
   ],
   "source": [
    "# 全部参数\n",
    "trainingNN(#1 训练迭代参数\n",
    "           trainingIterations = 13000,batchSize = 100,\n",
    "           #2 网络结构参数\n",
    "           numHiddenUnits1 = 200,numHiddenUnits2 = 80,inputSize = 784,numClasses = 10,               \n",
    "           #3 初始化参数\n",
    "           initConstant = 0.1, stddevConstant = 0.1, InitializationMethod = 'gauss',               \n",
    "           #4 激活函数参数\n",
    "           activationFunction = 'relu',\n",
    "           #5 正则参数\n",
    "           RegularExpression = 'l2',  regular_lambda = 0.0001,\n",
    "           #6 学习率参数\n",
    "           learning_rate = 0.4,global_step=tf.Variable(0),decay_steps = 150,decay_rate = 0.98,staircase=True,\n",
    "           #7 损失函数参数\n",
    "           loss_calculation = 'softmax'\n",
    "           )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Hint：\n",
    "- 多隐层\n",
    "- 激活函数\n",
    "- 正则化\n",
    "- 初始化\n",
    "- 摸索一下各个超参数\n",
    "  - 隐层神经元数量\n",
    "  - 学习率\n",
    "  - 正则化惩罚因子\n",
    "  - 最好每隔几个step就对loss、accuracy等等进行一次输出，有根据地进行调整"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 作业小结\n",
    "1. 学习了tensorflow的常量、变量、session、run等概念，对tensorflow的程序设计思路和运行流程有了一定了解，tensorflow机制很有特点\n",
    "2. 学习了神经网络(NN)的层级结构(隐层、神经元等)、初始化、激活函数、学习率、正则、损失函数等内容，参数调优仍然是个很大的工程\n",
    "3. 数据可视化采用了打印监测量的方式，便于捕捉训练过程的数值变化，获取更多细节，及时调整训练策略\n",
    "4. 七类参数封装了函数，统一调用，便于逐类认识和快速调优，努力编写清晰的程序结构，有助于提高效率\n",
    "5. 本次训练找极值采用各影响因子多次轮询分析的方法，没有用gridSearch的方法，这可能导致找到的极值是局部极点，有待提高运算力，优先使用多个参数维度同时遍历找全局最优的方法，以获得更好效果。\n",
    "\n",
    "\n",
    "name_space"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
